Skip to content

branch-3.0: [fix](load)fix load json format boolean type to int column. #54397#54640

Merged
dataroaring merged 1 commit intobranch-3.0from
auto-pick-54397-branch-3.0
Aug 14, 2025
Merged

branch-3.0: [fix](load)fix load json format boolean type to int column. #54397#54640
dataroaring merged 1 commit intobranch-3.0from
auto-pick-54397-branch-3.0

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 12, 2025

Cherry-picked from #54397

@hubgeter
[fix](load)fix load json format boolean type to int column. (#54397)
ed92ce7 
### What problem does this PR solve?

Related PR: #43469
Problem Summary:
PR #43469 accidentally removed the logic for reading boolean
"true"/"false" values in the simd join reader. Before PR #43469,
"true"/"false" were treated as "1"/"0", allowing a bool column in a JSON
file to be imported into an Doris int column. This PR restores this
logic.
@github-actions github-actions bot requested a review from dataroaring as a code owner August 12, 2025 22:52
@Thearas
Copy link
Contributor

Thearas commented Aug 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Aug 12, 2025
@Thearas
Copy link
Contributor

Thearas commented Aug 12, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 12, 2025

TPC-H: Total hot run time: 39879 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed92ce72db72593931be08c7a0122093d682639b, data reload: false

------ Round 1 ----------------------------------
q1	17597	7574	6693	6693
q2	2075	165	160	160
q3	10545	1150	1227	1150
q4	10206	737	764	737
q5	7739	2921	2883	2883
q6	216	135	134	134
q7	1013	630	610	610
q8	9368	1981	2040	1981
q9	6654	6376	6361	6361
q10	6991	2239	2302	2239
q11	457	264	261	261
q12	406	218	210	210
q13	17768	3013	3052	3013
q14	229	211	210	210
q15	518	471	462	462
q16	473	397	376	376
q17	997	633	527	527
q18	7259	6616	6642	6616
q19	1396	1119	963	963
q20	487	204	219	204
q21	3949	3217	3115	3115
q22	1098	974	979	974
Total cold run time: 107441 ms
Total hot run time: 39879 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6631	6577	6570	6570
q2	328	239	236	236
q3	2963	2949	2944	2944
q4	2079	1873	1878	1873
q5	5727	5759	5757	5757
q6	205	126	127	126
q7	2229	1822	1875	1822
q8	3398	3571	3616	3571
q9	8859	8997	8919	8919
q10	3589	3516	3524	3516
q11	603	497	497	497
q12	766	596	585	585
q13	8887	3194	3169	3169
q14	289	281	268	268
q15	509	478	476	476
q16	490	446	438	438
q17	1859	1624	1620	1620
q18	8221	7766	7744	7744
q19	1656	1492	1542	1492
q20	2074	1860	1903	1860
q21	5458	5315	5285	5285
q22	1167	1112	1048	1048
Total cold run time: 67987 ms
Total hot run time: 59816 ms

@doris-robot
Copy link

doris-robot commented Aug 12, 2025

TPC-DS: Total hot run time: 197366 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed92ce72db72593931be08c7a0122093d682639b, data reload: false

query1	1293	960	902	902
query2	6227	1925	1893	1893
query3	10954	4491	4516	4491
query4	33073	23526	23535	23526
query5	3436	454	440	440
query6	259	174	178	174
query7	3994	308	318	308
query8	282	259	238	238
query9	9463	2558	2572	2558
query10	472	267	260	260
query11	18010	15462	15283	15283
query12	163	100	100	100
query13	1558	444	430	430
query14	9333	7302	7194	7194
query15	250	191	191	191
query16	8093	514	525	514
query17	1665	625	602	602
query18	2152	332	321	321
query19	324	166	174	166
query20	138	117	113	113
query21	210	110	115	110
query22	4735	4460	4513	4460
query23	34901	34018	34191	34018
query24	11554	2957	2987	2957
query25	669	435	433	433
query26	1249	175	174	174
query27	2858	358	354	354
query28	7723	2188	2171	2171
query29	890	481	480	480
query30	303	168	160	160
query31	1057	803	838	803
query32	91	57	56	56
query33	767	295	295	295
query34	1170	500	515	500
query35	871	741	730	730
query36	1093	936	981	936
query37	129	67	69	67
query38	4134	3988	3948	3948
query39	1549	1462	1468	1462
query40	202	101	103	101
query41	49	50	50	50
query42	123	104	107	104
query43	523	483	485	483
query44	1241	810	817	810
query45	186	174	180	174
query46	1186	769	751	751
query47	2008	1870	1931	1870
query48	482	404	394	394
query49	953	418	433	418
query50	819	427	422	422
query51	7392	7225	7335	7225
query52	106	94	97	94
query53	274	197	187	187
query54	1366	488	486	486
query55	76	79	81	79
query56	272	252	249	249
query57	1324	1195	1218	1195
query58	237	232	218	218
query59	3105	3013	2975	2975
query60	301	272	274	272
query61	113	111	114	111
query62	889	690	704	690
query63	225	197	198	197
query64	5160	670	638	638
query65	3563	3308	3272	3272
query66	1324	300	298	298
query67	15954	15664	15453	15453
query68	4718	574	564	564
query69	456	264	267	264
query70	1187	1089	1038	1038
query71	348	263	257	257
query72	6194	3956	4121	3956
query73	762	344	352	344
query74	10512	8973	9374	8973
query75	3373	2665	2672	2665
query76	2666	1078	1062	1062
query77	385	277	268	268
query78	10443	9489	9554	9489
query79	2441	608	598	598
query80	880	421	428	421
query81	533	229	224	224
query82	682	91	88	88
query83	250	145	143	143
query84	228	82	81	81
query85	1564	305	287	287
query86	458	290	302	290
query87	4373	4234	4282	4234
query88	4129	2396	2351	2351
query89	414	292	298	292
query90	1959	186	188	186
query91	179	145	150	145
query92	61	50	52	50
query93	2317	561	557	557
query94	782	310	302	302
query95	370	267	269	267
query96	625	283	289	283
query97	3294	3166	3165	3165
query98	219	198	201	198
query99	1497	1294	1290	1290
Total cold run time: 304844 ms
Total hot run time: 197366 ms

@doris-robot
Copy link

doris-robot commented Aug 12, 2025

ClickBench: Total hot run time: 29.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed92ce72db72593931be08c7a0122093d682639b, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.11
query5	0.52	0.52	0.51
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.57	0.51	0.50
query10	0.58	0.56	0.56
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.61	0.61
query14	0.78	0.80	0.81
query15	0.85	0.83	0.82
query16	0.40	0.37	0.39
query17	1.06	1.04	1.07
query18	0.24	0.23	0.23
query19	1.91	1.82	1.80
query20	0.01	0.01	0.01
query21	15.41	0.58	0.60
query22	2.29	2.07	1.22
query23	17.15	1.01	0.75
query24	3.55	1.56	1.15
query25	0.22	0.08	0.05
query26	0.60	0.14	0.14
query27	0.05	0.04	0.05
query28	9.44	0.54	0.46
query29	12.59	3.24	3.20
query30	0.25	0.07	0.06
query31	2.86	0.40	0.40
query32	3.23	0.47	0.46
query33	3.00	3.01	3.03
query34	17.08	4.52	4.58
query35	4.56	4.63	4.54
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.39 s
Total hot run time: 29.8 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 12, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 42.07% (11185/26589)
Line Coverage 32.56% (95896/294479)
Region Coverage 31.71% (49576/156319)
Branch Coverage 28.18% (25447/90302)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 80.00% (8/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.00% (19683/26243)
Line Coverage 68.24% (200715/294119)
Region Coverage 66.35% (120136/181055)
Branch Coverage 59.68% (60935/102098)

dataroaring
dataroaring approved these changes Aug 13, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor Author

github-actions bot commented Aug 13, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 13, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Aug 13, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Aug 14, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 27f74f7 into branch-3.0 Aug 14, 2025
24 of 28 checks passed
@github-actions github-actions bot deleted the auto-pick-54397-branch-3.0 branch August 14, 2025 03:45
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @hello-stephen @dataroaring @hubgeter